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Abstract 


Research work in WP5 advances the state of the art in sanitization techniques, considering approaches 
for the anonymization of data based on k-anonymity and differential privacy. One goal of WPS5 is to pro- 
vide techniques for scalable use of sanitization techniques in data markets (i.e., by data owners and data 
scientists, in addition to privacy experts). This deliverable focuses on the use of metrics for the improved 
application of differential privacy in a Machine Learning scenario. To this end, we see high potential in 
privacy metrics that wrap anonymization parameters and relate to lawmaker requirements, and ease pa- 
rameter setting for data scientists and data owners. In this deliverable we will illustrate how differential 
privacy anonymization parameters (€,6) can be transformed into identifiability metrics, especially in ma- 
chine learning. The use of differential privacy during the training phase of a machine learning model offers 
a scalable anonymization technique. However, selecting the privacy parameters for differential privacy is a 
challenging task for data scientists since the actual strength of the parameters is dataset dependent. One po- 
tential remedy is to derive privacy parameters from semantic metrics. Here, membership inference attacks 
have received a lot of attention in the context of machine learning. However, membership inference attacks 
are strictly weaker than the attacks against which differential privacy protects, and thus privacy parameters 
chosen under membership inference will likely be too high. We formulate two identifiability bounds for the 
differential privacy adversary and show that these bounds can actually be reached. We are optimistic that 
these bounds can support data scientists in choosing privacy parameters, and that the bounds derive more 
efficient privacy parameters in comparison to previous work. In comparison to using membership inference 
attacks for measuring the strength of privacy parameters, our bounds are for the strong adversary assumed 
by DP and thus almost tight. 
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Executive Summary 


Differential privacy allows data scientists to bound the influence that members in the training 
data have on a machine learning model. To use differential privacy in machine learning with the 
differentially private stochastic gradient descent, data scientists must choose privacy parameters, 
e.g. (€,6) for the Gaussian mechanism. Values for (€,6) are difficult to choose because these 
worst-case upper bounds might not be tight for practical datasets. Concrete membership inference 
attacks have been used to choose €, but represent an empirically observed lower bound. Dif- 
ferential privacy aims to protect against adversaries with arbitrary auxiliary information, so only 
adversaries stronger than the membership inference adversary can lead to empirically verifiable 
upper bounds. Furthermore, the privacy parameters (€,6) do not match societal norms and legal 
requirements w.r.t. factual identifiability of the underlying training data when receiving a classifi- 
cation from a differentially private machine learning model. 

We put forward the idea of inferring privacy parameters (€, 6) based on an adversary’s Bayesian 
belief about the presence of a specific record in the training dataset. We bound the posterior belief 
for multidimensional queries under advanced composition, and observe that this bound can actu- 
ally be tight in practice. Furthermore, we connect the strong adversary considered by differential 
privacy to membership inference bounds by deriving a membership advantage bound. Posterior 
belief and expected membership advantage are derived directly from the differential privacy defi- 
nition and protect against the strong adversary with arbitrary auxiliary knowledge about all but one 
record in a database. Therefore, data owners can choose (€, ô) based on two tight identifiability- 
based metrics, namely maximum posterior belief and expected membership advantage. 


1. Introduction 


The research work in WP5 advances the state of the art in sanitization techniques, considering 
approaches for the anonymization of data based on k-anonymity and differential privacy. One goal 
of WP5 is to provide techniques for scalable use of sanitization techniques in data markets (i.e., 
by data owners and data scientists, in addition to privacy experts). This deliverable focuses on the 
use of metrics for the improved application of differential privacy in a Machine Learning scenario. 

Differential privacy (DP) has received much attention by the privacy research community, 
leading to key contributions such as the tight estimation of privacy loss under composition [[Miv17] 
and mechanisms for differentially private deep learning [ACG* 16]. However, data scientists have 
to choose privacy parameter (€,6) for training a machine learning model with the differentially 
private stochastic gradient descent. Several approaches for choosing and interpreting privacy pa- 
rameters have been introduced, yet they do not reflect identifiability (AS19] [HGH* 14], part from 
the original differential privacy definition or lack applica- 
bility to common differential privacy mechanisms [LCI1]. Especially in deep learning, practi- 
cal membership inference (MI) attacks have been used to measure identifiability (e.g.,[BGRK19| 
[YGFJ18]}). However, membership in- 
ference adversaries do not have unconstrained auxiliary information. DP adversaries are assumed 
to possess unconstrained auxiliary information and DP thus provides an upper bound on a strong 
adversary with background information about all data points in the input data but one. Therefore, 
MI attacks offer intuition about the outcome of practical attackers, but bounds on MI attacks in 
terms of differential privacy £ are not tight [JE19], and thus MI can only represent a lower bound 
on identifiability. 

Furthermore, since differential privacy aims to encourage data owners to participate in research 
studies that benefit society, intuitive communication of risk to individuals will strongly affect its 
widespread implementation [Nis16]. Data owners will likely only agree to offer their data to a 
research study or as training data for a machine learning model, if they can confidently assess 
their privacy protection. However, the factual identifiability risk is only indirectly specified by DP 
privacy parameters (£, ô) [NW18]. In fact, if data owners are told their answers to a survey are 
guaranteed to change the outcome only very slightly (i.e., by a factor ef) participation in the survey 
may even decrease, since data owners feel their contribution is not important [OK20]. Further- 
more, some privacy regulations refer to individual identifiability as a measure for 
anonymization, a concept that cannot directly be mapped to DP privacy parameters (e.g., Clifton et 
al. [(CT13]). In consequence, if DP is used to comply with privacy regulations in- 
terpreting the factual guarantees w.r.t. identifiability risk for privacy parameters (€, ô) is required. 

We formulate identifiability bounds and transform these bounds into concrete privacy param- 
eters (€,6). Rather than analyzing the MI adversary, we consider a Differential Identifiability 
adversary with unconstrained auxiliary knowledge and derive the maximum posterior belief Pg 
of the adversary as a worst case Bayesian identifiability bound related to (€,6). Furthermore, 
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we define the complementary metric of expected membership advantage pa. Expected member- 
ship advantage offers a quantification of the adversary’s probability of correctly identifying the 
underlying dataset D or D’. Concretely, the expected membership advantage specifies how of- 
ten posterior belief is greater than 50% and allows us a direct comparison with the membership 
advantage of Yeom et al. for the membership inference adversary. 

A subsequent question is whether our identifiability bounds are actually tight. Holding the 
distribution of differential privacy noise addition constant, multiple global (£, ô) guarantees will 
result from various chosen sensitivities, which quantify the difference between data sets that is 
covered by noise [NRSO7]. Therefore, the factual guarantee (¢€,6) depends on the difference 
between data sets and the identifiability bounds might accordingly not be tight. In differentially 
private deep learning, noise is scaled to the difference between possible gradients; however, the 
estimated global sensitivity might far exceed the factual sensitivity, since the training data records 
are likely to be within the same domain (e.g., pictures of cars vs. pictures of nature scenes). We 
propose scaling the sensitivity to the difference between the gradients of a fixed data set and any 
neighboring dataset to achieve a tight bound. We evaluate how tight our identifiability bounds are 
for one data analytics and one machine learning reference data set, and show that we can indeed 
achieve tight bounds. Our main contributions are: 


e Formulate identifiability bounds for the posterior belief and the expected membership ad- 
vantage that can be transformed into privacy parameters (€,6) and used in conjunction with 
composition. 


e The practical implementation of an adversary that meets all assumptions of worst-case ad- 
versaries against DP. 


e A heuristic for scaling sensitivity of the differentially private stochastic gradient descent. 
This heuristic leads to tight bounds. 


This deliverable is structured as follows. Preliminaries are presented in Chapter [2| where 
we provide an overview of notations and concepts that are used throughout this deliverable. Af- 
terwards, we formulate identifiability metrics and analyze their upper bounds in Chapter [3] and 
Chapter f] The metrics are evaluated for a reference dataset from the deep learning domain in 
Chapter[5] We present conclusions in Chapter|6] 
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2. Preliminaries 


In the following three sections we will present the building blocks that are used and extended 
in this deliverable. Section [2.1] first provides fundamentals with respect to differential privacy, 
mechanisms for differentially private machine learning and composition. Secondly, Section [2.2] 
and Section [2.3] introduce membership inference and differential identifiability experiments that 
we will use for comparison between the DP adversary and the MI adversary throughout this work. 


2.1 Differential Privacy 


We define data analysis as the evaluation of a function f : DOM — R on a dataset D from domain 
DOM yielding a result r from the set of all results R. DOM is assumed to be a finite set, and 
D consists of independently sampled values from a distribution over DOM (LQS* 13}. Since r is 
computed from D, r inevitably leaks information about the respective entries d € D (cf. impossi- 
bility of Dalenius’ desideratum [Dwo06]). Differential Privacy offers an anonymization 
guarantee for statistical functions such as those found in data analysis. In contrast to semantic 
anonymization (e.g., k-anonymity [SS98|[Sam01]), DP perturbs the result of a function f(-) over 
a dataset D = {d\,...,dn} s.t. the result of f(-) could have been produced from dataset D or some 
neighboring dataset D’. A neighboring dataset D’ either differs in the presence of one data point 
from D (unbounded DP) or in the value of one data point in D (bounded DP). Thus, plausible de- 
niability is provided to participants in the dataset D since their impact on the query function f(-) 
becomes bounded. DP provides a strong guarantee: protection against an adversary with knowl- 
edge of all points in a data set except one. However, one assumption of DP is that data points 
are independent; correlation between data points, such as those that may be found in social graph 
applications, cannot yield the same DP guarantees [KMI1]. To add differentially private noise to 
the result of some arbitrary function f(-), mechanisms M according to Definition [I]are used. In 
the context of this work, we will assume w.l.o.g. that D \ D’ Æ {}; in other words, D contains one 
datapoint more than D’ when unbounded DP is considered. 


Definition 1 ((€,5)-Differential Privacy [DKM*06]). A mechanism M gives (€,5)-Differential 
Privacy if for all independently sampled D,D’ C DOM, where DOM is a finite set, differing in at 
most one element, and all possible mechanism outputs S 


Pr(M(D) € S) < e€ Pr(M(D') E€ S) +8 


We define £-DP as (€,6 = 0)-DP and refer to the application of a mechanism M to a func- 
tion f(-) as output perturbation. The Gaussian mechanism is the predominant DP mechanism in 
machine learning for perturbing the outcome of stochastic gradient descent, and adds noise inde- 
pendently sampled from a Gaussian distribution centered at zero. Prior work has analyzed 
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the tails of the normal distributions and found that bounding the standard deviation as follows 


fulfills (€, ô) DP. 
o > Afoy/21n(1.25/5) /e (2.1) 


o depends not only on the DP guarantee, but also on a scaling factor Af. DP holds if mech- 
anisms are scaled to the global sensitivity Af of Definition [2| i.e., the maximum contribution of 
a record in the dataset to the outcome of f(-). For example, in the case of counting queries Af 
is usually 1, while for the sum of all salaries in a company Af might be very large (e.g., reflect- 
ing the CEO salary). The DP guarantee is tight for any data point having an influence of Af. 
Let D and D’ be neighboring data sets, the global /2-sensitivity of a function f is defined as 


Af = maxp.p'||f(P) — f(D’) Ila. 


Definition 2 (Global Sensitivity). Let D and D' be neighboring. For a given finite set DOM and 
function f the global sensitivity Af with respect to a distance function is 


Af =max||f(D) — f(D’) 


Note that the absolute global sensitivity as in Definition [2|can also be defined relative to lo- 
cal sensitivity as Af = max LS '~(D). The impact of local sensitivity, compared to using global 
sensitivity, is that less noise is added when € is held constant, and € is decreased when the noise 
distribution is held constant. Local sensitivity is specified in Definition B] [NRSO7] and scales the 
differential privacy protection to a fixed dataset D. 


Definition 3 (Local Sensitivity). Let D and D' be neighboring. For a given finite set DOM, inde- 
pendently sampled dataset D C DOM, and function f, the local sensitivity LS ¢(D) with respect to 
a distance function is 


LS;(D) = max || f(D) — f(D’) 


In the differentially private stochastic gradient descent, perturbed outputs are released repeat- 
edly in an iterative process. The most basic form of composition for accounting of multiple data 
releases is sequential composition, which states that for a sequence of k mechanism executions 
each providing (£;, 6;)-DP, the total privacy guarantee composes to (X; €, ¥; 6;)-DP. However, 
sequential composition adds more noise than necessary. A tighter analysis of composition is 
provided by Mironov [Miv17]: (@,€rpp)—Rényi Differential Privacy (RDP) describes the dif- 
ference in distributions M (D), M(D’) by their Rényi divergence [VEH1O]. For a sequence of k 
mechanism executions each providing (œ, Erppi)-RDP, the total privacy guarantee composes to 
(a, L; €rppi)-RDP. The (œ, €rpp)-RDP guarantee converts to (€gpp — a, 6)-DP. The Gaussian 
mechanism is calibrated to RDP using the relation Egpp = a-A f / 20°. 


2.2 Membership Inference 


Membership inference is a threat model for quantifying identifiability in machine learning. MI 
attacks are used to quantify how accurate an adversary can identify members of training data. 
Black-box MI assumes access to a trained machine learning model [SSSS17|JE19], and white-box 
MI extends the assumption to oberservations on the training data [NSH1I8]. Yeom et al. 
formalize MI in the following generic adversarial experiment: 
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Experiment 1. (Membership Inference Exp™') Let Ay, be an adversary, M be a differentially 
private learning algorithm, n be a positive integer, and Dist be a distribution over data points 
(x,y). Sample D ~ Dist” and let F = M(D). The membership experiment proceeds as follows: 


1. Sample zp uniformly from D and Zpist from Dist 


2. Choose b + {0,1} uniformly at random 


fe ifb=1 
Z = 


3. Let 


ZDist if b=0 


4. Amı outputs b' = Ayr(7,z,Dist,n,M) € {0,1}. Ifb' =b, Amı succeeds and the output of 
the experiment is I, it is O otherwise 


Within this deliverable we aim to evaluate white-box attacks consistent with DP guaran- 
tees (i.e., auxiliary side knowledge). We thus extended the black-box experiment from Yeom 
et al. to consider not only mechanism outputs, but also the mechanism and, implicitly, 
the privacy parameters (€, ô) themselves. This addition solely provides Amz with additional infor- 
mation, so any upper bounds for the probability to succeed in this white-box experiment will hold 
for the black-box experiment. The probability of success in the above experiment is bound by the 
DP privacy parameter € [YGFJ18]; however, the bound is very loose in practice [JE19]. 


2.3 Differential Identifiability 


Lee et al. introduce Differential Identifiability as a strong threat model for inferring 
membership in the input dataset of a function based on perturbed output. Differential Identifi- 
ability assumes the adversary to calculate the likelihood of all possible input datasets, so called 
possible worlds in a set ¥, given a mechanism output. Li et al. show that the Dif- 
ferential Identifiability threat model maps to the worst case against which bounded Differential 
Privacy protects when |¥| = 2, since DP considers two neighboring datasets D, D’ by definition, 
and possible worlds each have the same number of records. The DI adversary Apr knows both 
neighboring datasets and receives the multidimensional output 7 of the mechanism applied to one 
of these two datasets. The adversary’s task is to guess which of the two datasets D’ or D was cho- 
sen as input. The experiment is similar to membership inference, since the attacker must decide 
whether the dataset contains the member that differs between the known D’ and D or not. To allow 
us comparisons, we reformulate the original idea as a cryptographic experiment: 


Experiment 2. (Differential Identifiability Exp?!) Let Apt be an adversary, M be a differentially 
private learning algorithm, D and D' be neighboring data sets drawn mutually independently from 
distribution Dist, using either bounded or unbounded definitions. The Differential Identifiability 
experiment Exp”! proceeds as follows: 


1. Set fp := M(D) and Fp := M(D") 
2. Choose b + {0,1} uniformly at random 


3. Let 


_ (7p, ifb=1 
r= 
7p, ifb=0 
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4. Apr outputs b! = Apr (f, D, D',m ,Dist) € {0,1}. Ifb! =b, Apr succeeds and the output of 


the experiment is 1, it is O otherwise 


Since this experiment precisely defines an adversary with access to background knowledge on 
D and D’, Anz is an implementable instance of the DP adversary [DR16]. 
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3. Identifiability bounds for the differential 
privacy adversary in Machine Learning 


In this chapter we formulate two bounds on identifiability of individual training records when 
releasing a differentially private machine learning model. The bounds hold for differential privacy 
under multidimensional queries and composition. While membership inference is commonly used 
for quantifying identifiability in machine learning, we consider the stronger DP adversary Apr. 
We prove that protection against Apr also protects against Ayr (Section B-1). Afterwards, we 
introduce our identifiability bounds for Apr (Section (3.2). First, we define the identifiability risk 
as adaptive posterior belief, which is a new privacy metric for iterative mechanisms. Second, we 
discuss membership advantage for Apr, which is a privacy metric complementing the adaptive 
posterior belief by offering a scaled quantification of the adversary’s probability of success (i.e., 
how often posterior belief is greater than 50%). 


3.1 Identifiability under the Strong Probabilistic Adversary 


Societal norms such as identifiability w.r.t. anonymization are not matching to the mathemati- 
cal concept behind DP, since DP limits the contribution of an individual to aggregated informa- 
tion INWIS]. For example, the European General Data Protection Regulation (GDPR) Recital 
26 states that identifiability is determined by all means reasonably likely to be used to single 
out an individual and the American Health Insurance Portability and Accountability Act 
(HIPAA) explicitly requires identifiability guarantees in form of group sizes (i.e., 1/group size) 
in § 164.514 (2) [Ame10]. Rather, in a move from DP guarantees to societal norms, privacy 
parameter £ should be transformed to the probability of identifiability (CT13}(CK12]. 

We propose the strong probabilistic adversary Apr as an alternative to using the MI adversary 
for evaluating DP guarantees IRRLMI18]. DP 
aims to protect against adversaries with arbitrary auxiliary information, so intuitively, MI bounds 
based on DP guarantees will never be reached. The results of Jayaraman et al. confirm this 
expectation empirically, citing a large gap between the theoretical membership advantage bound 
formulated by DP (eë — 1) and the empirical membership advantage, which implies that more pow- 
erful inference attacks exist. Apr performs such a stronger attack, which can also be implemented 
and yields metrics related to identifiability. Apr has access to arbitrary auxiliary information as is 
assumed in the original DP guarantee. Ap; also operates in a white-box model and consequently 
observes all training steps of a machine learning algorithm, a characteristic especially found in 
federated learning. Therefore, Apr quantifies what the strongest possible DP adversary can infer. 
We first show that protection against DI implies protection against MI. Equivalently, we show that 
Apr is stronger than Amz due to the additional available auxiliary information. Concretely, the 
Apr knows both neighboring data sets D and D’ instead of only receiving one value z and the size 
n of the data set from which the data points are drawn. 
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A Abt 
Mr D! e Dist”! z+ Dist, D := D' Uz, b + {0,1} 4 
i ifb=0, r:=M(D) else r := M(D' 
ub y n:=|D| r,D,D! x nt (D) elser (D') 
z:=D\D' M pist 
z,r,n,Dist 
op pt S 
if z is in the dataset producing r b! := b" if b' =b (Apr wins) return | 


b" := 1 else b” := 0 


Figure 3.1: Reduction of Apr to Amz, shown here for unbounded DP 


Proposition 1. Differential identifiability implies membership inference: if Amı wins Expyy, then 
one can construct Ap; that wins Exppy, as shown in Figure|3. 1] 


Proof. We prove the proposition by contradiction: assume that the mechanism M successfully 
protects against Apr, but that there exists an adversary Amz that wins Expy;. Again, we assume 
w.Lo.g. that D\ D’ Æ {}; otherwise, we can formulate an analogous proof. We construct an 
adversary Ap; that also wins Expp, as follows, see also Figure[3.1} 


1. On inputs D, D', M,F, Dist, Apr calculates n = |D] and let z = D \ D’. 
2. Apr gives (z,7,n,Dist) to Amz. 

3. Amı gives b” = Ayj;(z,7,n,Dist) to Ap; in response. 

4. Ap; outputs b. 


By the definition of Expp;, Apz wins if b’ = b, and thus succeeds in the following cases. 

Case 1: b = 1, which means F = M(D). Since z € D, this is exactly the case where Amz correctly 
outputs b’ = 1. Therefore b’ = b. 

Case 2: b = 0, which means F = M(D’). Since z ¢ D’, this is exactly the case where Amr 
correctly outputs b’ = 0. Therefore b' = b. For both cases Apr wins (b! = b), which contradicts 
the assumption that the mechanism M successfully protects against Apr. It is more difficult for a 
mechanism to protect against Expp, than against Expy, which is equivalent to the statement that 


if Amı wins Expy;, then Apr wins Expp, as well. 


3.2 Posterior Belief and Advantage 


We will now introduce two identifiability metrics for the differentially private gradient descent 
under Apr. In line with Li et al. we also assume that datasets D and D’ to be sampled 
mutually independently from the identical distribution (i.i.d.) Dist over datasets. Under this 
assumption the samples of the dataset are essentially sampled independently, however, potentially 
with different probabilities (i.e., stratified). 
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Under this assumption the samples of the dataset are essentially sampled independently, how- 
ever, potentially with different probabilities. Note that this assumption includes not only the usual 
i.i.d. assumption, but also stratified sampling. 

Here we model these algorithms as iterative algorithms where each unperturbed gradient is 
modeled as a function evaluation of the form F41 = f(D,7) = f,(D). In the case of gradient 
descent we update the weights F using the perturbed gradients g, = M,(D,7,_1) and learning rate 
a as ras = F, + a-&,. The initial value can be modeled as ro, and the functions and the mechanism 
M may thus change during the iterations. This iterative procedure requires an extension of Apr 
to composition, where Apr receives a matrix R = 70,...,77 that consists of the results from all 
training steps. In this deliverable we will analyze these steps under DP composition. 

We quantify individual identifiability by the Bayesian posterior belief in the right dataset D 
compared to the belief in D’. Deriving a tight upper bound for the posterior belief of Apr in 
Section |4. I]then results in the highest chance of successful inference of an individual. 


Definition 4 (Adaptive Posterior Belief). Consider the setting of Experiment |2|and denote R; = 
(70,71,--.,Fi) as the result matrix, comprising i multidimensional mechanism results. The poste- 
rior belief in the correct dataset D is defined as the probability conditional on all the information 
observed during the adaptive computations 


Pr(D, Rx) 


By=Pr(D|Rx) = Pr(D, Rx) + Pr(D’, Rx) 


where the probability Pr(D|R,) is over the random iterative choices of the mechanisms up to step 
k. 


While the posterior belief is in principle a sophisticated probability distribution, it is now 
shown that the iterative procedure leads to a significant simplification: each B; can be computed 
from the previous f;_;. It turns out, that the final belief can be computed using the following 
Lemma{I|which we will use later to further analyze the strongest possible attacker Ap; of Experi- 


ment 


Lemma 1 (Calculation of the posterior belief). Assuming uniform prior and independent mecha- 
nism M; (more precisely, the noise of the mechanisms must be sampled independently) the poste- 
rior belief on dataset D can be computed as 


= Th, Pr(Mi(D) =F) 
> TR PrMi(D) =F) + PMD) =F) 


Be 


Proof. We prove the lemma by iteration over k. 

k= 1: We assume the attacker starts with uniform priors Pr(D) = Pr(D’) = 4. Thus, B;(D|R1) 
can be directly calculated using the definition and division of both numerator and denominator by 
the numerator: 


D|R\) = 
Bi (PIRI) = SRD) =A) HPM (D) =F) 
1 
~ 7) PAM DNA) 
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k—1-+k: Inthe second step B,_;(D|R,_1) is used as the prior, using the shorthand notations fy := 
By(D|Rx), and in the last step py := Pr(M,(D) = Fx) and pi, := Pr(M;(D’) = F) the calculation 
of B.(D|R_) starts as for the induction start k = 1 


Pr(M;(D) = 7x) - Pk- 1 
Pr(My(D) = Fk) - Be—-1 +Pr(M:(D') = Fk): (1 — Br-1) 
1 
1+ Pr(M;(D!)=7) —Pr(Mi(D)=Fx)- Bri 
Pr(Mi(D) =F): Bk-1 
1 
1 F P- Pbk- Pi Br 1 


-Pebe 


B= 


Now the induction assumption can be substituted for o right term of the denominator and 


then multiplying the numerator and denominator with We í Dit We q p; leads to 


1 me | Di 
— pi Br-1 _ Pk PLIE 1p HET 
prbr-1 Tet Pi 
Pky 1 Pit HE | pt 
(joy Pi + TS Pi) — PLT 
_ Pk i=1 Pi t i=1 Pi Ph i=1 Pi 
p [IE Pi 
_ Mhe iP 
TE, Pi 


where in the last step the first and the third term in the denominator cancel out and lead to the 


desired result when inserted back into the last form of B, above. 


The above proof illustrates that Apr behaves as a binary classifier that chooses the option 
w.r.t. to the highest posterior probability. Specifically, Apr computes posterior beliefs B(-) for 
datasets D and D’ and guesses the dataset 


argmax B(D|R;). 

DeE{D,D'} 
This decision process can be simplified as follows: the probabilistic mechanism M turns Rx into 
random variables which are denoted as 


Xı := M(D) and Xo := M(D') (3.1) 


corresponding to the cases b = 1 and b = 0, respectively. Since Ap; knows D,D’ and the mecha- 
nism M, Apr also knows the corresponding probability densities gy, and gx,. The densities have 
the same shape depending on the mechanism but are centered at the different unperturbed results 
f(D) and f(D’), respectively, as visualized in Figure[3.2(a)] with f(D) =0, f(D’) = 1 for multi- 
ple DP guarantees. Assuming uniform prior beliefs, Apz’s decision then depends on whether Apr 
believes that R, stems more likely from X; or Xo and therefore decides 


Api (Rr, D, D’, M, Dist) = arg max gy, (Re) (3.2) 

be{0,1} 
More generically, if we choose not to assume uniform prior beliefs, Apr instead chooses the 
dataset that results in a larger posterior belief. The posterior belief in our simple example is 
visualized in Figure|3.2(b)| So Apr is essentially a naive Bayes classifier whose decision boundary 
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Figure 3.2: The decision boundary of Ap; does not change when increasing the privacy guarantee 
since (£, ô) causes the PDFs of D and D’ to become squeezed. Thus Apr will exclusively choose 
D if a value is sampled from the left, red region, and vice versa for D’ in the right, blue region. 
Still, confidence towards either decision declines. 


is depicted by the change in background color in Figure[3.2(b)| The input features are the perturbed 
results Rz, and the exact probability distributions of each class is known. Note that the distributions 
are entirely defined by D, D’, and the mechanism M, so Apr does not use the knowledge of the 
distribution Dist from which D and D’ were sampled. 

The posterior belief quantifies the probability for the original dataset D for a single optimiza- 
tion procedure resulting in the specific results R; however, in another optimization instance the 
result Rg could differ. In Section|4.1] we will therefore define an upper bound on B(D). 

We expect the question “How likely is it that an adversary correctly guessed the data set in 
which I have participated?” to be a major point of interest when interpreting DP guarantees in 
iterative evaluations of M, like those found in data science use cases such as machine learning. 
The bound pg for the posterior belief indicates the worst case probability of identifying that a 
given, single record belongs to the dataset D. When posterior belief pg is low, an individual can 
plausibly deny that the hypothesis of Ap; is correct. In practice, it may be even more important to 
know how often Ap; makes a correct hypothesis over the course of many runs, which only occurs 
when pg > 50%. This is quantified by the advantage, which is the success rate normalized to the 
range |—1, 1], where 0 corresponds to random guessing. 


Definition 5 (Advantage). Consider an experiment Exp and denote R; = (f0,71,.--,7;) as the 
result matrix, comprising i multidimensional mechanism results. The advantage is then defined as 
Adv=2Pr(Exp = 1)-1 (3.3) 


where the probability is over the random iterative choices of the mechanisms up to step k. When 
we consider Exp”!, we define the corresponding advantage as Adv?!. When we consider Exp™', 
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we define the corresponding advantage as Adv“", 


In Section [4.2] tight upper bounds for the advantage will be derived. The advantage and its 
upper bounds may be important ingredients for the technical implementation of legal privacy re- 
quirements that formulate anonymization breaches in terms of individual identifiability. 
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4. Derivation of tight upper bounds 


Within this Chapter we will derive tight upper bounds for the previously introduced identifiabil- 
ity metrics posterior belief and advantage in Section [4.1] and Section [4.2] Afterwards, we will 
illustrate how the bounds can be transformed into privacy parameters under RDP composition in 
Section 


4.1 Posterior Belief Bound 


Bayesian posterior belief was introduced as a measurement of identifiability. We formulate a 
generic bound on posterior belief independent of datasets D and D’, the mechanism M, and the 
mechanism output matrix R, which consists of multiple multidimensional mechanism outputs 7;. 
The proposed bound solely assumes that the DP bound holds and makes no further simplifications 
or assumptions, which results in an identifiability-based interpretation of DP guarantees. 

Theorem [2] shows that Apr operates under the sequential composition theorem. The sequen- 
tial composition theorem states that given M; providing ¢;-Differential Privacy, the sequence of 
Mı 
Differential Privacy but also for (€,6)-Differential Privacy setting which is common when using 
the differentially private stochastic gradient descent in the machine learning setting [ACG* 16}. 


geen Qh I J Re Pee’ B Lassy 


Theorem 2 (Bounds for the Adaptive Posterior Belief). Consider experiment Exp”! with neigh- 
boring datasets D and D'. 

(i) Let M,,..., Mx be a sequence of arbitrary but independent differentially private learning al- 
gorithms providing €,...,€,-Differential Privacy to functions f; with multidimensional output. 
Then the strong probabilistic adversary’s posterior belief is bounded by 


D|Rx) < pp=———>-— 
Be(P|Rx) < Pp jee 


(ii) Let M,..., Mx be a sequence of arbitrary but independent differentially private learning 
algorithms where each M; provides (€;,6;)-Differential Privacy to multidimensional functions fi. 
Then the same bound as above holds with probability 1 — ¥%_; 6. 


Proof. (i) The adversary with unbiased prior (i.e., 0.5) regarding neighboring datasets D,D’ has 
a maximum posterior belief of 1/(1-+e~*) when the €-differentially private Laplace mechanism 
is applied to a function with a scalar output [LC12]. This upper bound holds also for arbitrary 
€-differentially private learning algorithms with multidimensional output. We bound the general 
belief calculation by the inequality of Definition [I] According to DP, for any differentially private 
mechanism result 7, where differentially private mechanism M (D) has been trained with dataset 
D and M(D') has been trained with dataset D’: 
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Pr(M(D) =F) < e€ Pr(M(D’) =7) +6 


Assuming equal priors, the posterior belief can be calculated as follows: 


1 
D|R) = 
PDR) 1 DE PMD) 
IŻ; Pr(M;:(D)=7;) 
< i : m 
1 Ti- Pr(Mi(D/) =") 


For 6 = 0, the last equation simplifies to: 


1 
Ty Pr(Mi(D’)=7) 
Ties eêi Pr(Mi(D')=7)) 
Lo 
1+ je 14e- Eh t 


P (DIR) < i 


Equivalently, one can specify a desired posterior belief and calculate the overall €, which can 
be spent on a composition of differentially private queries: 


e=m( PE) 
1— pg 


The value for 6 can be chosen independently according to the recommendation that 6 << t 
with N points in the input dataset. 


4.2 Bound for Expected Membership Advantage 


Now we derive the upper bound for the advantage of Ap; (Adv?!). Membership advantage is 
used to quantify the success of membership inference adversaries, and bounds in terms of DP € 
have been previously derived for MI [YGFJ18]. First we derive that this general bound for Amy 
also holds for Apr, as expected based on Proposition [I] We then derive a tighter bound for the 
(€,6)-differentially private Gaussian mechanism, which is commonly used for machine learning, 
by modeling Ap; as a naive Bayes classifier. 


Proposition 2 (General Bound on the Expected Adversarial Membership Advantage). For any 
€-DP mechanism, the identification advantage of Apr in experiment Exp”! can be bounded as 


Adv?! < (e£ —1)Pr( Apr = 1|b = 0) (4.1) 


Proof. First the definition is re-written using both ways to success in the experiment. Then using 
that both datasets are chosen equally likely by the challenger (Pr(b = 1) = Pr(b = 0) = 1/2), 
substituting Pr(b’ = 0 |b = 0) by the probability of the complementary event 1 — Pr(b’ = 1 |b =0)) 
and finally substituting b’ = 1 by Apr = 1 leads to the formula of Yeom et al. 


Adv?! = 2(Pr(b = 1)Pr(b! = 1 |b = 1)+ 
+Pr(b =0)Pr(b' =0|b=0))-1 
= Pr( Apr = 1|b = 1) — Pr( Apr = 1|b = 0) (4.2) 
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which is the difference between the probability for true detection of D minus the probability of 
incorrectly choosing D. Now we use the fact that the mechanism M turns r into random variables 
Xı := M(D) and Xo := M(D") for the cases b = 1 and b = 0, respectively. We formulate the 
probability density functions as gy, and gy,. Additionally A(r) is introduced as a shorthand for 
Ap1(7,D,D', M, Dist) 


Adv?! = Pr(Apr = I|r = M(D)) —Pr( Ar = Ilr = M(D’)) 


= Lp p) (Anr (F, D, D', M ,Dist))— 

E,- M(D ae D,D’,M,Dist)) 
= i sx AG) dr— f sx (AGA (43) 
= | (ex,@)— gx *)A War (4.4) 


Since €-DP is defined as Pr(M(D) € S) < e€Pr(M(D’) € S), it yields to the same inequality 
gx, < eê gx, for the densities for all S (i.e., at each point), 
vPI < (ef —1) f sx (AC AF 


ef —1 


Bounding Pr(Apz = 1|b = 0) by 1 results in Adv?! < eë — 1. Since the mechanism preserves 
some utility, a rational adversary Ap; that tries to win will make a correct guess at least 50% of the 
time, so Pr( Apr = 1|b = 0) < 0.5. Substituting this into Equation yields to a tighter bound 
Adv?! < (e€ — 1)/2. 

When .Ap; acts as a naive Bayes classifier, only a complete lack of utility from infinite noise 
results in Pr( Apr = 1|b =0) =0.5. Otherwise, Pr( Apr = 1|b = 0) is far smaller than 0.5; therefore, 
even this membership advantage bound is usually not tight. Since protection against DI implies 
protection against MI, as proven in Proposition [I] the bound also holds for Amz. A similar bound 
for advantage has been proven for the MI adversary [YGFJ18], e€ — 1; however, (e€ — 1)/2 is 
smaller and will therefore also be tighter. Apr should also come closer than Amz to the bound. 
This is in line with Jayaraman et al. who expect that this would be the case for a stronger 
inference attack than MI. 

We now derive a tighter bound p4 on Adv” for the Gaussian mechanism and (£, 6)-differential 
privacy, continuing from Equation (4.4). Note that under n assumption of equal priors, the 
strongest possible adversary of Equation ( maximizes (4.4) by choosing b = 1 if (gx, (7) — 
8x, (7)) > 0 and b = 0 otherwise. The SUNE bound on eo is constructed from Apy’s strategy; 
however, it holds for all weaker adversaries, including Amz. Since we argue that Ap; precisely 
represents the assumptions of DP, the bound should hold for other possible attacks in the realm of 
DP and the Gaussian mechanism under the i.i.d. assumption. 

Now, since Ap; is essentially a naive Bayes classifier with known probability distributions, we 
can use the properties of normal distributions (we refer to Tumer et al. for full details). 
Looking at the decision boundary of this classifier (i.e., when to choose D or D’) under M Gau 
with different (€, ô) guarantees, we find that the decision boundary does not change as long as the 
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Figure 4.1: For visualization purposes, we arbitrarily set f(D) = 0, f(D’) = 1. The plots show 
Apr error regions for varying £, MGau, f(D), f(D’). Note that the probability density functions 
and thus the regions under the curve are not scaled by the prior. 


probability density functions (PDF) are symmetric. For example, consider the given datasets D, D’ 
and mechanism M : DOM —> R that yield f(D) =0 and f(D’) = 1 without noise. Furthermore, 
assuming w.l.o.g. that Af = 1. If a (6, 10~°)-DP M Gau is applied for perturbation, Ap; has 
to choose between the two PDFs in Figure [4.1(a)] based on the output M(-) =r. The regions 
where Apr chooses D are shaded red in both figures, and regions that result in the choice D’ are 
shaded blue. Increasing the privacy guarantee to (3, 10~°)-DP in Figure|4.1(b)| squeezes the PDFs 
and confidence curves. However, the decision boundary of the regions at which Ap; chooses a 
certain dataset stay the same. Thus, it is important to note that holding r constant and reducing 
(€,5) solely affects the posterior belief of Apr, not the choice (i.e., the order from most to least 
confident is maintained even while maximum posterior belief is lowered). The corresponding 
regions of error are shaded in Figures [4.1(a)| and |4.1(b)| where we see that a stronger guarantee 
reduces Adv”, 


We assumed throughout our work that Ap; has uniform prior beliefs on the possible databases 
D and D’ over which f() was evaluated. This distribution is iteratively updated based on the pos- 
terior resulting from the mechanism output r. Thus, Ap; is essentially representing a Bayesian 
classifier. This allows us to analyze the concrete distributions resulting from the upper bounds pg 
and pa and the mechanism M. If, for example, Moa, is applied to achieve (€,5)-DP, we can 
determine the expected membership advantage of the practical attacker Apr analytically by the 
overlap of the resulting Gaussian distributions p. 321]. We thus consider two multidi- 
mensional Gaussian PDFs (i.e., M(D), M(D’)) with covariance matrix £ and means (without 


noise) HW; = f(D), fb = f(D’). 


Theorem 3 (Tight Bound on the Expected Adversarial Membership Advantage). For the (€,6)- 
differentially private Gaussian mechanism, the expected membership advantage of the strong prob- 
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abilistic adversary on either data set D, D'. 


Adv?! < p,=2® a 
2,/2In(1.25/65) 


where ® is the cumulative density function of the standard normal distribution. 


Proof. The starting point for the derivation is Equation where the Gauss-distributions are 
to be inserted for gy, and gx,- Since both distributions arise from the same mechanism, they 
have the same È but different means yı = f(D) and uo = f(D’). Since the strongest adversary 
is the Bayes adversary that chooses according to Equation @B.2), and we assume equal priors, the 
decision boundary between D and D’ is the point of intersection of the densities (see Figure/4.1(a)| 
for the 1D-case). In general, this is exactly the situation of linear discriminant analysis where 
it is known to be a hyperplane halfway between uı = f(D) and Uo = f(D’). Mathematically, 
by setting In(gx,) = In(gx,), the plane can be calculated to be halfway (A/2) between the two 
centers, where A is the Mahalanobis distance A := y(i — fh)? =~! (i — fh). Notably 
the decision boundary between D and D’ notably does not depend on È and therefore €, but the 


distance between f(D) and f(D’) G.e., sensitivity). As we add independent noise in all dimensions 
£ = o°], integration in Equation (4.3) normal to this direction leads to factors 1 and only the 1-D 
integration along the direction through the 2 centers remains with A simplified to l al, Thus, 


Adv?! = @(A/2) — ®(—A/2) 
=2@(A/2)-1 


=29 (IA) 


20; 


Inserting the standard deviation needed for (£, ô)-DP from then yields 
Ras ( | = Hallo 


2A f2(./2In(1.25/5)/e) 


<f i ) 1=pa 
2(./2In(1.25/8)) 


The theorem can be used to calculate € from a chosen maximum expected advantage 
1 
e = \/21n(1.25/5) (=) 


As for the posterior belief, the (£, ô) guarantees with ô > 0 can be expressed via a scalar value 
Pa. However, a specific membership advantage must be computed individually for different kinds 
of mechanisms M. We provide plots of pg and pa for different (€,6) in Figure For Pa, 
the curves are specific for MGau. In contrast, pg is independent of M. To compute both mea- 
sures, we use Theorem |2|and Theorem]3| We also assume w.l.o.g. that f(D) = (01,02,...,0;) and 
f(D’) = (11, 12,..., 14) for all dimensions k. Thus, f(D) and f(D’) are maximally distinguish- 
able, resulting in Afp = Vk. Figure|4.2(a)jillustrates that there is no significant difference between 
the adaptive posterior belief pg for €-DP and (€,6)-DP for 0 < 6 < 0.1. In contrast, p, strongly 
depends on the choice of 6 as depicted in Figure|4.2(b)| For example, pa is low for (2, 10~°)-DP 
indicating that the probability of Apr choosing D is similar to choosing D’. Yet, the corresponding 
Pp is high, which provides support that Apr guesses is correct. 
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Figure 4.2: The expected adversarial worst-case confidence bound pg and the adversarial mem- 
bership advantage p4 for various (€, ô) when using M Gau for perturbation. 


4.3 Bounds under composition with Renyi Differential Privacy 


In practice, data owners can choose an overall privacy guarantee (€,6) according to values for the 
bounds pg and pg. In iterative settings, such as ML, the data owner will have to perform multiple 
mechanism executions, which necessitates the use of composition theorems to split the total guar- 
antee into guarantees per iteration (&;, ô). Theorem [2] provides a bound for posterior belief B for 
DP under sequential composition, so, for k steps the data owner can simply divide £; = €/k and 
6; = 6/k, confirming that Apz’s belief composes as expected by sequential composition. How- 
ever, sequential composition only offers loose bounds in practice [DRV10}|KOV17]. In addition, 
Theorem|3]states the bound pg on membership advantage for a single mechanism execution with- 
out considering composition. We suggest to compose mechanisms with RDP composition, which 
permits a tight analysis of the privacy loss over a series of mechanisms. Therefore, we adapt both 
Pp and Pq to RDP. 

(Maximum Posterior Belief) We first demonstrate that RDP composition results in stronger 
(€,6) guarantees than sequential composition for a fixed bound pg: 


DIR) = 4.6 
Bx( | ) 1 IÉ Pr(M;:(D')=F;) (4.6) 
Ti; Pr(M;(D)=F;) 
1 
= T+ Ths e~ (Exppi+(a-1)7! In(1/6;)) i) 
7 1 
ye ek(a—1)-!In(1/8)) LE Err; 
B 1 
~ qpel% 17 In(1/8K)—-X$; ere; 
k 
= $ erori — (& — 1)" In(1/5#) = pp (4.8) 
i=l 


Equation implies that an RDP-composed bound can be achieved with a composed 6 
equal to 6*. We know that sequential composition results in a composed 6 value equal to kô. 
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Since ô% < kô, RDP offers a stronger (€,5) guarantee for the same Pg, and results in a tighter 
bound for pg under composition. This behavior can also be interpreted as the fact that holding 
the composed (€, ô) guarantee constant, the value of pg is greater when sequential composition is 
used compared to RDP. 

(Expected Membership Advantage) A similar analysis of the expected membership advan- 
tage under composition is required when considering a series of mechanism M and function 
f. We restrict our elucidations to the Gaussian mechanism. The k-fold composition of M Gau;; 
each step guaranteeing (Œ, €gpp;)-RDP, can be represented by a single execution of Mga, with 
k-dimensional output guaranteeing (œ, €gpp = k€gppi)-RDP. To prove this, we bound || U1 ; — H2, || 
for each of the composed mechanism executions by A f2. Theorem [3] yields 


Adv?! = 2@(A/2)—1 


29 (Ile Bk) i 


K Viti = pill Hotei 


2A fo /a/(2€rpp;) 2€rDPi) 


= ; \/a/(2érvpi) ae) 
(EE) 
VE) 


The result shows that Ap; fully takes advantage of the RDP composition properties of Egppi 


and œ. As expected, p, takes on the same value, regardless of whether k composition steps with 
Erpp; Or a Single composition step with Erpp is carried out. Therefore, we can calculate the final 
Pa for processes with multiple iterations, like the training of deep learning models, and a desired 
final p, can be broken down into a privacy guarantee per composition step with RDP. 
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5. Application to deep learning using DPSGD 


In DP, the use of global sensitivity Af often results in a mechanism that yields unnecessarily high 
noise, which does not reflect the function’s insensitivity to individual inputs [NRSO7]. Because € 
is often not a tight bound under global sensitivity, Nissim et al. proposed local sensitivity, 
which depends not only on the function, but also on the input data to be used. Local sensitivity 
no longer protects against inference on any possible adjacent datasets, but only on the chosen true 
dataset D and any dataset adjacent to it. The local sensitivity approach decreases noise addition by 
narrowing the guarantee, while still protecting the true dataset used for a calculation. We suggest 
a heuristic for estimating local sensitivity in a deep learning setting and investigate the impacts 
of this local sensitivity estimate on posterior belief and membership advantage. Here, a neural 
network (NN) is provided a training dataset D to learn a prediction function $ = fan(¥) given 
(x,y) € D. Learning is achieved by means of an optimizer and a variety of differentially private 
optimizers for deep learning are availabld!| These optimizers represent a differentially private 
training algorithm that updates the weights 0, per training step t € T with 0 + 0-1; —Q@-&, 
where œ > 0 is the learning rate and g, = M,(D) denotes the Gaussian perturbed gradient. After 
T update steps, where each update step is itself an application of the Gaussian mechanism, the 
algorithm outputs a differentially private weight matrix Or which is then used in the prediction 
function frn(-). Considering the evaluation of fnn(-) given (x,y) € D as post-processing of the 
trained weights Or, we find that prediction ĵ = fnn(X) is (€,6)-differentially private too. 

We assume that Ap; desires to correctly identify the dataset with the correct representation of 
a record d when having the choice between D and D' that differ in d. Furthermore, Apr is assumed 
to possess the initial weights 6o, the perturbed gradients g, after every epoch, the values of privacy 
parameters (€,6), and sensitivity Af, = C equal to the clipping norm. There are two variations 
of DP: bounded and unbounded. In bounded DP, it holds that |D| = |D’|, which was the standard 
in this work so far. Differentially private deep learning optimizers such as the one utilized in this 
work consider unbounded DP as the standard case in which |D| — |D’| = 1, whereas 
we considered bounded DP until now. We continue to consider bounded DP, but we will extend 
our arguments and experiments to unbounded DP within this section. In both cases, D and D’ 
are independently sampled from a distribution in line with Definition [I] In machine learning, the 
finite set DOM from which D and D’ are sampled is the total available dataset. The datapoints not 
sampled will result in a test set, which can be used to evaluate the utility of the trained model. 

In this section, A fz refers to the sensitivity with which noise added by a mechanism is scaled, 
not necessarily global sensitivity. In some experiments, for example, we set Af. = LS, (D) with 
average clipped gradient calculation g;, using local sensitivity as in Definition B} rather than global 
sensitivity as in Definition [2] The assumptions are very similar to those of white-box membership 
inference attacks for federated learning [NSHI18]. In federated learning, multiple participants, 


‘All experiments within our work were realized by using the Tensorflow privacy package: https: //github.com/ 
tensorflow/privacy 
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each with their own private training data, jointly train a global model by sharing gradients for their 
subsets of training data with an aggregator, who combines the gradients and shares the aggregated 
update with all participants. In the role of either aggregator or participant, the adversary observes 
the updates of the model, as assumed in our experiments. The attack is defined in Algorithm [I] 
To implement bounded DP with global sensitivity, D’ contains n — 1 records and Af) = 2C, since 
the maximum influence of one example on the sum of per-example gradients is C. If one record 
is replaced with another, the lengths of the clipped gradients of these two records could each be C 
and point in opposite directions, which results in n- ||8,(D’) — 8;(D)||2 = 2C. We also note that the 
same value of A fz used by Ap; must also be used by M to add noise. 

The motivation behind Algorithm [I] is intuitively as follows. The initial weights 6) and the 
clipping norm C can be thought of as constants in the gradient functions %9(D) and 89(D’), which 
also depend on cost function J. Apr computes these gradient values based on J,C, 69, D, and D’ 
and then compares them to the perturbed gradient gp to determine posterior belief B,. Apr applies 
& to @ with knowledge of œ to receive @, and consequently repeats the cycle for all epochs, 
updating the posterior belief at every step. 


Algorithm 1 Strong Adaptive Adversary in Deep Learning 


Require: Datasets D and D’ with n and n — 1 records D; and D’;, respectively, training steps T, 
cost function J(@), perturbed gradients g, for each training step t < T, initial weights 0, prior 
beliefs By(D) = Bo(D’) = 0.5, learning rate œ, clipping threshold C, and mechanism M 

Ensure: Adversary Confidence pr(D) 

1: for t € [T] do Compute gradients 

: For each i € D,D’, compute g;(D;) — Vo-J(@, Di) and g: (D'i) — Vos (0, D'i) 

: Clip gradients 

: Clip each g,(D;),g;(D’;) for i € D,D’ to have a maximum Z? norm C using g,(D;) + 

ai(D;)/max(1, "Pal and g, (D';) — g, (D'i) /max(1, lal) 

5: Calculate Batch gradients 

6: 8(D) « avg(8(Di)) 

7: (D)  avg(g(D',)) 

8 

9 


kW N 


: Calculate Sensitivity 


: Af, HC 
10: Calculate Belief 
3 B:(D)*Pr(M (8 (D))=&1] 
L: Brii(D) — gome (D)) =e B (D Pr MED) =e 


Compute weights 
12: O41 & 0 — ag; 
13: end for 


A motivating factor for MI attacks is that strong DP guarantees are difficult to achieve without 
significant utility loss in deep learning settings [JE19||[BPS19]. This utility loss occurs because the 
DP guarantee protects all possible datasets, although only the training data itself must be protected 
in deep learning. Clipping norm C bounds the influence of a single training example on training by 
clipping each per-example gradient to the chosen value of C. Although this value bounds the influ- 
ence of a single example on the gradient, this bound is loose, since it does not necessarily reflect 
the difference between the training dataset and possible neighboring datasets. Thus, the gradients 
may be far from C. In DP, when sensitivity is set to a value larger than necessary, the guarantee € 
is not reached, so our metrics p, and pg will not be reached either. If n- ||g;(D) — 8;(D’)|| < C, 
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adversary confidence B(D) would be very small in every case when Afo = C, which is the case in 
most implementations of differentially private neural networks. This scenario can be thought of as 
using a global sensitivity of C, rather than local sensitivity. We suggest addressing this by fixing the 
dataset D and considering only datasets D’ adjacent to this fixed D; however, approximating local 
sensitivity for neural network training is difficult because the gradient function output depends not 
only on D and D’, but also on the architecture and current weights of the network. To avoid this 
dilemma, we propose a metric, dataset sensitivity, in Definition|6]with which we strive to consider 
the D’ with the largest difference to D within the ML dataset in an effort to approximate local 
sensitivity. 


Definition 6 (Dataset Sensitivity). For a given ML dataset U and neighboring datasets D, D' C 
U the dataset sensitivity DS(D) with respect to a distance function is 


DS(D) = max||D—P'| 


The motivation behind Definition [6] is based on the assumption that local sensitivity can be 
approximated as LSz,(D) = n-||%:(D) — &,(D’)|| where DS(D) = ||D — D'||. The simplification 
from local sensitivity to dataset sensitivity allows us to bypass the complex gradient calculations. 
Instead of scaling noise to an arbitrarily chosen C, for which we show in Figure[5.1(b)]that it is not 
necessarily tight, we calculate DS(D) in order to identify D'. Based on our assumption, we can 
then scale noise to the approximated value of LSs,(D) = n- ||%:(D) — &:(D’)|| and achieve local 
sensitivity for any weights and architecture. Our experimental observations further confirmed this 
expectation. 

This procedure makes our choice of D indistinguishable from any D’ within the chosen range 
of possible values. For our experiments, this range of values is the entire Modified National 
Institute of Standards and Technology database (MNIST) of handwritten digits, and D and D’ each 
contain 100 data points. We choose the data points x; and x2 in MNIST that result in the highest 
dataset sensitivity DS(D), with xı € D and x2 ¢ D. D’ is then formed by replacing xı with x2 in 
D. We measure similarity for the dataset sensitivity with the structural similarity index measure 
(SSIMP| which allows us to examine individual pairs of datapoints instead of the entire datasets 
because ||D — D'|| = ||x1 —x2||. Since different images will result in very different gradients, we 
can calculate DS(D) = ||x; — x2|| and approximate LS, (D) = n- ||8(D) — &(D’)|| = |ê x1) — 
&;(x2)||. For unbounded DP, we remove the data point with the smallest SSIM distance to all other 
images from D to form D’. 

Based on the previously introduced assumptions and notations we adapt Ap; to local sen- 
sitivity. The implementation of Ap; for the differentially private stochastic gradient descent is 
stated in Algorithm |1| and specifies Ap; in an unbounded environment with global sensitivity. 
For bounded DP with local sensitivity, Algorithm |I}can be adjusted, s.t. D’ is fixed to n training 
records, and Af, = LS3,(D) = n - ||&:(D’) — &:(D)||2.. To implement unbounded DP with local 
sensitivity, A f2 =n-||n- 8,(D’) — (n—1)- &:(D)||2 and D’ contains n — 1 records. 


5.1 Setting of the experiment 


For practical evaluation, we build a feed-forward NN for the MNIST dataset] For MNIST our 
NN architecture consists of two repetitions of a convolutional layer with kernel size (3,3), batch 


‘https://ece.uwaterloo.ca/~z70wang/research/ssim/ 
Overview and detailed description available at: http: //yann.lecun.com/exdb/mnist/ 
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Figure 5.1: Sensitivity and posterior belief (30 epochs) for pg = 0.9 and 6 = 0.01 
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Figure 5.3: Distribution of n - ||8,(D) — 8,(D’)|| from max to min difference in D and D’ 
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Local Af Global Af 
Bounded DP (0.239,0.002) (0.194,0) 
Unbounded DP (0.228,0.002) (0.273,0.004) 


Table 5.1: Empirical advantage (a, 5) and empirical 6 which is within 6 


normalization and max pooling with pool size (2,2) before being flattened for the output layer. 
We use relu and softmax activation functions for the convolutional layers and the output layer, 
respectively. 

One epoch represents the evaluation of all records in D. Thus, it is important to highlight 
that the number of update steps T varies in practice depending on the number of records from 
D used for calculating the DP gradient update g. In mini-batch gradient descent a number of b 
records from D is used for calculating an update and one epoch results in t = D/b update steps. 
In contrast in batch gradient descent all records in D are used for calculating the update and each 
epoch consists of a single update step. While all approaches vary in their speed of convergence 
due to the gradient update behavior (i.e., many small updates vs. few large updates) none of the 
approaches has hard limitations w.r.t. convergence of accuracy and loss. Within our work, we 
operate with batch gradient descent and given differentially private gradient updates g¢ after any 
update step t Ap; decides whether g was calculated on D or D’. We assume that Ap; has equal 
prior beliefs of 0.5 on D and D’. The prior belief of Apr adapts at every step t. 

In the experiments, relevant parameters are set as follows: training data |D| = 100, epochs 
k = 30, clipping norm C = 3.0, learning rate œ = 0.005, 6 = 0.01, and pg = 0.9. The overall 
privacy parameter €, transformed from pg, is split into a per step privacy parameter according to 
RDP composition. We choose D randomly and select D’ to maximize dataset sensitivity. This 
choice of D’ represents the case in which the bounds will be most closely reached, since the 
sensitivity between f(D)and f(D’) is actually reached. In practice, during private deep learning, 
& must be calculated also for D’ if local sensitivity is to be estimated; otherwise, g on D’ does 
not have to be computed. We merely do so in order to empirically validate Ap; during training. 
We run experiments for both local and global sensitivity to evaluate the effect of the sensitivity on 
identifiability and utility, and we evaluate both bounded and unbounded settings. For each of the 
four cases, we train a neural network and simulate Apr 1000 times. We then analyze the resulting 
distribution of posterior beliefs and calculate the membership advantage by counting the cases in 
which posterior belief for D exceeds 0.5. 


5.2 Results 


We present the empirically calculated values (i.e., after the training) for pa = 0.2562 and 6 in 
Table|5. i] The belief distributions for the described experiments can be found in Figure 
Note that although Figure [5.1(a)] shows posterior belief Br(D) exceeding pg in some cases, 
Table|5.1|confirms that 6 indeed bounds the percentage of experiments for which Br (D) > pg. For 
all experiments with local sensitivity and for global, unbounded DP, the empirical and analytical 
values of pa match the empirical values. However, in global, bounded differential privacy the 
difference of correct guesses and incorrect guesses by Ap; falls below pa. The percentage of 
evaluation runs for which Br(D) > pg is also far lower. This behavior confirms the hypothesis 
that C is loose, so global sensitivity results in a lower value of B;(D), as is again confirmed by 


Figures|5.1(a)jand|5.1(b)} We also notice that the distributions in Figures|5.1(a)|for local sensitivit 
8 8 y 
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in bounded and unbounded settings look identical to each other. This observation confirms that the 
strong adaptive adversary attack model is applicable to chose privacy parameter € in deep learning. 

We now investigate the reason for the similarities between unbounded differential privacy with 
local and global sensitivity and also for the differences between Figures|5.1(a)|concerning bounded 
differential privacy with local and global sensitivity. In the unbounded case, the distributions seem 
identical, which occurs when Af = LS¢,(D) = ||(n— 1) - 8:(D’) —n- &:(D)||2 = C, so the clipped 
per example gradient of the differentiating example in D should have the length 3, which is equal to 
C. This hypothesis is confirmed with a glance at the development of ||(m — 1) -8,(D’) —n-8,(D)]2 
in Figure [5.1(b)| This behavior is not surprising, since all per example gradients over the course 
of all epochs were greater than or close to C = 3. In the bounded differential privacy experiments, 
Af = LS3,(D) =n: ||8:(D’) — 8:(D)||2 4 2C, since the corresponding distributions in Figure[5.1(a)| 
do not look identical. This expectation is confirmed by the plot of n - ||(D’) — 2:(D)||2 in Fig- 
ure|5.1(b)| This difference implies that the per example gradients of the differentiating examples 
in D’ and D are less than 2C and do not point in opposite directions. We also point out that the 
length of gradients tends to decrease over the course of training, so if training converges to a 
point in which gradients are shorter than the chosen value of C, globally differentially private deep 
learning inherently offers a stronger privacy guarantee than was originally chosen. 

A glance at Figure |5.2]confirms that the differentially trained models in these models do, in- 
deed, yield some utility. The visualized accuracy was achieved by increasing the training set size 
to 10,000. We also observe that test accuracy is directly affected by the value of sensitivity A fz 
chosen for noise addition. Since gradients in all four scenarios are clipped to the same value of < C, 
the only differences between training the neural networks is Af. As visualized in Figure [5.1(b)| 
global and local sensitivities for unbounded DP were identical, so the nearly identical correspond- 
ing distributions in Figure[5.2|do not come as a surprise. Similarly, we observe that A fz is greater 
for global, bounded DP in Figure [5.1(b)| so utility is also lower for this case in Figure [5.2] The 
unbounded DP case with local sensitivity yields the highest utility, which can be explained by the 
low value of Af) that can be read from Figure|5.1(b)} 

To confirm our claim that maximizing dataset sensitivity from Definition [6] allows us to ap- 
proximate local sensitivity, we carry out network training for 250 runs with differing choices of 
D'. We first evaluated the top three choices of D’ that maximize dataset sensitivity, then the three 
choices that minimize dataset sensitivity most. The resulting n - ||%(D) — %,(D’)|| can be read 
from Figure [5.3] We see that the choices which maximize dataset sensitivity result in larger val- 
ues, while choosing D’ to minimize dataset sensitivity results in a smaller value. We therefore see 
a downward trend from left to right in Figure[5.3] 
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6. Conclusions 


We defined two identifiability bounds for the strong DP adversary in Machine Learning with the 
differentially private stochastic gradient descent, maximum posterior belief pg and expected mem- 
bership advantage pa. These two bounds can be transformed to privacy parameters (€,6). In con- 
sequence, with p4 and pg, data owners and data scientists can map legal and social expectations 
towards identifiability to corresponding privacy parameters (€,6). Furthermore, we implemented 
an instance of the DP adversary for Machine Learning and showed that the bounds can be reached 
under multidimensional queries with composition. To reach the bound it is necessary that the 
sensitivity is reflecting the actual local sensitivity of the dataset. We approximate LS;(D) for 
stochastic gradient descent, improving the utility of the differentially private model training when 
compared to the use of global sensitivity Af and reaching the bounds. 

Within MOSAICrOWN, and in data markets after the project concluded, we see large potential 
for choosing the privacy parameter as a transformation of identifiability bounds due to two reasons. 
First, identifiability bounds are on a well-defined scale between 0.5 and 1.0 and therefore go 
along with probabilities that we encounter and assess already in our daily life. In contrast privacy 
parameter € is defined between 0 and positive infinity. Second, if paired with local sensitivity our 
bounds can actually be reached and thus, higher privacy parameters € might be usable. This will 
likely have a positive effect on utility, which we illustrated in our evaluation. 
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